cleanup logic reworked - #1204

…ason

reworked cleanup implementation for lvol migration

…siting to cleanup target

…p monitor

…lt values and API parity params change (#1198)

* Add restart phase for mass create tests

* Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change

* Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change

* Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change

* Add K8s host-level core dump collection to e2e test framework

Previously, check_core_dump() only detected core dumps inside SPDK pods
at /etc/simplyblock/ but never copied them, and completely missed host-level
core dumps at /var/lib/systemd/coredump/. When SPDK crashed in K8s, all
crash evidence was silently lost after test runs.

Changes:
- k8s_utils.py: Add copy_core_dumps_from_spdk_pod() to kubectl cp core
  files from inside SPDK pods to the NFS log directory
- k8s_utils.py: Add collect_host_core_dumps() which uses the running SPDK
  pod (privileged, accesses host via /proc/1/root/) as primary path, with
  platform-aware fallback when SPDK pod is down:
  - OpenShift: oc debug node/ + chroot /host
  - Vanilla K8s: privileged pod with nsenter
  - Talos: privileged pod with hostPath volume mount (no host binaries needed)
- k8s_utils.py: Promote get_all_k8s_node_names() and detect_openshift()
  from continuous_k8s_native_failover.py to shared K8sUtils class
- cluster_test_base.py: Enhance check_core_dump() to actually copy pod core
  dumps and collect host-level core dumps via _check_host_core_dumps_k8s()
- continuous_k8s_native_failover.py: Refactor to delegate to K8sUtils methods

* Separate mass stress test into restart and no-restart scenarios

Existing mass create/delete tests had node outages (Phase 3b/7b) baked
into every run, conflating capacity testing with restart resilience.
This splits them into two distinct scenario types:

- Existing tests: ENABLE_NODE_OUTAGE defaults to False, so all current
  tests now run at full entity count without node restarts, enabling
  proper max-capacity delete testing.

- New restart variants: 6 new classes (Docker + K8s) with a 6000 total
  entity cap, node outages at Phase 3b/7b, and PERSISTENT_RETRY=True.
  Entity cap formula (MAX_ENTITY_COUNT // (1 + SNAPSHOTS_PER_LVOL))
  dynamically reduces lvol count per snapshot ratio:
    1 snap/lvol  → 3000 lvols
    6 snaps/lvol →  857 lvols
   10 snaps/lvol →  545 lvols

* Fix restore wait to fail test immediately on restore failure

_wait_for_restore_task_done had two bugs that masked restore failures:

1. except Exception swallowed AssertionError raised when task status was
   "failed", turning it into a warning log and continuing to poll until
   timeout. Added explicit re-raise for AssertionError.

2. After timeout expired, the method logged a warning and returned,
   letting callers proceed with checksum verification on incomplete
   restores — producing misleading "checksum mismatch" failures. Now
   asserts immediately on timeout.

Also adds missing non-restart mass stress test variants:
- MassCreateDeletePersistent_300x10_6Snap_K8s
- MassCreateDeletePersistent_300x10_10Snap_K8s
- MassCreateDeletePersistent_300x10_10Snap_Docker

* Capping clone creates

* Capping clone creates

* Fix K8s mass stress test: cleanup deleting infra PVCs, FIO leak, stall timeouts

- Scope PVC cleanup to test prefixes only (mcd-pvc-*, clone-pvc-*).
  The cleanup was deleting ALL PVCs in the namespace including
  simplyblock infrastructure PVCs, destabilising the cluster.
  Remove the kubectl delete pvc --all fallback on timeout.

- Delete Phase 6 FIO Jobs before Phase 7 clone PVC deletion.
  FIO pods left running block PVC finalizer removal, causing
  50 PVCs to stall for 300s+ and fail the test.

- Add DELETE_STALL_TIMEOUT (600s) separate from BOUND_STALL_TIMEOUT
  for delete verification. CSI provisioner processes PV deletions
  serially (~1.5s/vol), so 300s is too short at 850+ PVCs.

- Fix backend verification to call sbcli_utils.list_lvols() and
  sbcli_utils.list_snapshots() instead of k8s_utils which lacks
  these methods, causing AttributeError and skipping verification.

* Increases max entity to 9000

* Fix K8s mass create cleanup: respect deadline and preserve_resources_on_failure

- Increase MAX_TEST_DURATION from 6h to 10h for K8s tests
- Increase CLEANUP_TIMEOUT from 30min to 1h
- All cleanup steps now check deadline before running
- Replace unbounded sbcli_utils.delete_all_snapshots() with
  deadline-aware _cleanup_delete_backend_snapshots_with_deadline()
- Increase volumesnapshot batch delete size from 50 to 200
- Skip internal cleanup when preserve_resources_on_failure is set
  and test has failed (soft failure or exception)

* Fix upgrade test infrastructure: migration script, version detection, helm release

- Docker R25→R26 migration script: add missing mini lvol/snapshot re-write
  steps that existed in K8s version and UPGRADE.md but were absent in Docker
- Docker _is_r25_to_r26_upgrade(): fix detection to handle 'main' as target
  branch (trigger when base is R25 and target is NOT R25, instead of
  requiring target to start with 'r26')
- K8s HELM_RELEASE_SBCLI: change default from empty string to 'sbcli' so
  the R25 sbcli chart is actually uninstalled during R25→R26 migration
- K8s workflow: export HELM_RELEASE_SBCLI env var to test execution step
  with conditional value based on upgrade_type

* Update K8s upgrade workflow: r25_base_config default to remove_snode_init_container

The p2p-migration branch is outdated; remove_snode_init_container is the
current R25 branch used for K8s R25→R26 upgrade bootstrapping.

* Update K8s workflow image defaults: docker.io → ECR/Docker Hub shorthand

Replace outdated docker.io/simplyblock/ registry prefixes across all K8s
workflow files to match the current convention (topology-suite pattern):
- simplyblock_repository: public.ecr.aws/simply-block/simplyblock
- operator_repository: simplyblock/simplyblock-operator
- csi_repository: simplyblock/spdkcsi
- spdk_image: simplyblock/spdk:main-latest

Affected workflows: k8s-native-e2e, k8s-native-stress, k8s-native-e2e-add-node,
k8s-native-e2e-node-migration, monitoring-suite-k8s-native, k8s-native-upgrade.

* Align upgrade workflow defaults with e2e/stress pipelines

- Fix IP defaults: 192.168.10.211 → .210 (mgmt), 205-208 → 201-204 (storage)
- Remove NR_HUGEPAGES input (hardcode '2048' in env, override via EXTRA_SN_ARGS)
- Fix BOOTSTRAP_DATA_CHUNKS default: 2 → 1 (match e2e/stress)
- Fix BOOTSTRAP_ENABLE_NODE_AFFINITY default: false → true
- Add missing inputs: EXTRA_CLUSTER_ARGS, EXTRA_SN_ARGS, CLUSTER_SECURITY
- Add "Write cluster security/backup config" step before bootstrap
- Add EXTRA_CLUSTER_ARGS/EXTRA_SN_ARGS support to bootstrap step

Applied to: upgrade-bootstrap.yml, upgrade-bootstrap-single.yml,
upgrade-bootstrap-single-v2.yml.

* Fix upgrade test: mandatory target images, unconditional DB migration, pip error handling

- Make --target_spdk_image and --target_docker_image required in upgrade_e2e.py
  to prevent silent failures when images are empty
- Rename _is_r25_to_r26_upgrade() to _needs_db_migration() and run migration
  for all cross-version upgrades (not just R25→R26)
- Add raise_on_error=True to _pip_install_target so pip failures are caught
- Add default TARGET_SPDK_IMAGE and TARGET_DOCKER_IMAGE env vars in all 3
  upgrade workflow files

* Fix Slack notification skip and add test names to pipeline run-name

- Add send_slack_notification to e2e-bootstrap.yml workflow_dispatch inputs
  (was only in workflow_call, causing null != false to skip Slack step)
- Add run-name with test class/case to e2e-bootstrap.yml and e2e-docker.yml

* Fix api-parity-audit: pass send_slack_notification and lower max-subsys to 40

* Fix api-parity-audit: register TestAPIParityAudit in ALL_TESTS and add parity keyword

- Add TestAPIParityAudit to ALL_TESTS list in e2e/__init__.py so the
  e2e runner can discover it
- Import get_parity_tests and add "parity" keyword handler in e2e.py
- Fix K8s backup merge detection in TestBackupPolicyVersionsOne: detect
  pruning by backup count decrease (K8s CRD) in addition to status field
  (Docker/sbcli mode)

* Add missing sbcli_utils_v2.py required by TestAPIParityAudit

The file was missed when test_api_parity_audit.py was originally
committed (e2ee3ba). The import fails in CI because the module
is not tracked. Force-added past the sbcli* gitignore pattern.

* Fix api-parity-audit: register TestAPIParityAudit in ALL_TESTS and add parity keyword

- Fix get_io_stats() missing cluster_id argument in cluster.iostats audit
- Add detailed summary output to test log (findings by category)
- Fail the test on error-level findings instead of silently passing
- Register TestAPIParityAudit in ALL_TESTS so --testname discovery works
- Disable lvol-level backup delete tests (SFAM-2792):
  TestBackupRetentionMergeAfterDelete, TestBackupDeleteAndRestore,
  TestBackupDeleteInProgress
- Revert SBCLI_BRANCH default back to 'main' (bootstrap uses it)

* Add parity report artifact upload to e2e pipeline

Copy api_parity_report.html and api_parity_findings.json from NFS
into sbcli/e2e/logs/parity_report/ so they are included in the
existing log artifact upload step.

* Disable backup delete operations in stress tests (SFAM-2792)

- BackupStressMarathon: remove delete_and_backup from weighted
  operations and comment out _do_delete_and_backup method
- BackupStressRetentionMergeCycles: disable entirely since its
  sole purpose is delete-merge-restore cycles

* Add parity audit findings summary to e2e-bootstrap job summary

When TEST_CLASS is TestAPIParityAudit, parse the api_parity_findings.json
sidecar and append a severity breakdown table (errors/warnings/info) to
the GitHub Actions job summary.

…l this process before it's completion

…d CVEs

The Trivy job in .github/workflows/security.yml was failing daily. Two
separate causes:

- docker/Dockerfile's `COPY . /app` was shipping the repo's local
  .tox/.mypy_cache/.ruff_cache/.pytest_cache dirs into the scanned image
  (.tox alone added ~900MB across several stale python3.9/3.12/3.13
  virtualenvs, each with its own old, genuinely vulnerable pip/wheel).
  None of these are excluded by .dockerignore, so every local tox/mypy/
  ruff run before a build fattens the shipped image with real CVEs.
- The remaining 3 findings (msgpack 1.1.2, setuptools 70.3.0) come from
  pip 26.2's own vendored bundle (pip/_vendor), not from anything we
  install; pip still ships those exact versions upstream, so there is
  no fix available from our side. Added .trivyignore.yaml to suppress
  just those 3 finding IDs, wired via the trivyignores input.

Verified locally end-to-end (build docker/Dockerfile, scan with
aquasec/trivy:latest replicating the workflow's flags) — image shrank
from 1.96GB to 1.09GB and the scan now exits 0 with zero findings.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fix: lvol migration getting killed and cleanup not running

…anup function

#1203)

calculate_core_allocations()'s colocate_lvs branch put lvol_poller_core
  on app_thread_core's slot. Since jc_singleton_core is assigned in every
  branch, storage_node_ops.py's poller-group RPC unconditionally
  preferred jc_singleton_mask (added in e3e8fd0 to force the two onto
  the same core) � which silently clobbered the >=32 vCPU tier's
  deliberately dedicated lvol_poller core every time, defeating the
  point of giving it one.

  Colocate with jc_singleton_core's slot instead, so lvol_poller_mask
  becomes the single source of truth: equal to jc_singleton_mask when
  nothing dedicated was assigned, distinct when it was. Update
  add_node()/_restart_storage_node_impl() to use lvol_poller_mask
  directly (jc_singleton_mask only as a last-resort fallback if that
  reservation came up empty) and fix their now-stale comments to match.

  Add regression coverage for calculate_core_allocations' colocation
  behavior across all three size tiers � there was none before.

…9 incident)

A healthy node was force-shut and restarted a second time: while its
restart task was still running on the parallel pool, the dispatch mode
flipped (fd_dead_recovery_allowed went false as the first domain peers
came back ONLINE) and the main loop's inline path — which consulted
neither _restart_inflight nor _node_inflight — re-entered the same task.
Every guard in the second entry was blinded by two lost updates from
full-object writes of stale in-memory copies: the defer path's
task.write_to_db un-canceled the task and wiped its owner lease, and
_persist_target_device_event reverted the node's committed
in_restart->online flip back to in_restart.

- tasks_runner_restart.main(): single dispatch path — every execution
  checks the inflight maps and registers its future; serialized mode
  submits identically and waits (fut.result()), so parallel<->inline
  mode flips are harmless in both directions.
- tasks_runner_restart: new _task_finish/_task_update helpers write
  tasks via db.atomic_update (CAS on the fresh row, updated_at lease
  stamp) — a write can no longer resurrect a concurrently canceled/done
  task or clobber its owner lease. All task_runner_node writes
  converted; a lost CAS means another actor owns the outcome and the
  runner stops. Plus a fresh task re-read immediately before the
  destructive shutdown step.
- distr_controller._persist_target_device_event: atomic_update whose
  mutator touches only the device entries — concurrent node status
  flips survive device-event fan-out during restart waves.
- storage_node_ops.shutdown_storage_node: honour the result of the
  final OFFLINE set_node_status instead of reporting success over a
  half-committed shutdown; deliberately NOT whitelisting the
  RESTARTING->OFFLINE flip (it would strand a genuine concurrent
  restart whose final ONLINE CAS then gets refused).
- shutdown_storage_node/check_node_shutdown_preconditions: thread
  current_restart_task_id (bare uuid) through so the restart runner's
  own cleanup shutdown is not reported as a competing restart task.
- tests/unit/tasks: fake db mirrors the atomic_update contract.

Follow-ups (not in this change): unconditional node-RESTARTING entry
guard + lease-aware watchdog skip, atomic _add_task creation (duplicate
task TOCTOU), gating the self-heal force-ONLINE branch, converting
task_runner_device writes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…method

Design rule: the control plane forces LVS leadership only where it is
structural and race-free — lvstore creation, activation, and the restart
flow's fenced demote->grant handoff (recreate_lvstore ### 5/### 7).
Recovery paths repair the redirect topology and wait for IO-driven
self-promotion; they never grant.

Incident 2026-07-30 (sb_logs_20260730_195000_30m, LVS_9): the guarded
last-resort grant fired from the lvol-migration runner 0.45s before the
dead primary's restart task was even created (the "no handoff task
active" guard raced task creation), seating the secondary as writer.
When the primary's restart later ran its own fenced demote->grant
handoff, writer_conflict events fired on jm_vuid=9 and the demoted
secondary's JM write lock lingered for its full ~64s lease
(jfi_r_wr_lock tms_delta=64103ms on both JMs), blocking the new
leader's journal writes for that window. Earlier motivations stand too:
a CP-forced grant outside the restart flow skips the primary's blob-md
reload (2026-07-06 LVS_13 stale metadata) and the run-20260725
grant/demote flapping.

- _recover_leaderless_lvs: keep the single-flight lock, hublvol repair
  and bounded self-promotion wait; drop the last-resort
  bdev_lvol_set_leader(leader=True) — still-leaderless now returns None
  and object operations keep failing fast (no_leader_cache) until IO
  promotes the primary or a restart re-places leadership.
- Remove the grant-only guard helpers _leadership_moving_tasks_active
  and _taker_jm_quorum_ok.
- tests: leaderless recovery must never call bdev_lvol_set_leader.

Open data-plane item (separate): the demote path does not release the
JM write lock, so every handoff — including IO-driven self-promotion —
waits out the ~64s lease before the new writer can write the journal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tion

The namespaced-subsystem pick (get_next_available_subsystem_on_node) and
the in_creation record write were separate reads/writes, so two concurrent
creates/clones could both count the same shared subsystem as having one
free namespace slot and both join it past capacity.

DBController.claim_lvol_ns_slot now runs pick + record write in ONE FDB
transaction: the record itself is the slot claim (occupancy is recounted
from lvol records inside the transaction). A per-node allocator key gives
concurrent claims a read conflict so the loser retries and recounts with
the winner's record present, while the lvol-table read itself is a
snapshot read (no conflict range over the whole table — unrelated lvol
writes must not abort claims). Wired into create_lvol, snapshot clone,
and the -32602 add-time fallback, which now also excludes the subsystem
SPDK just rejected instead of being able to re-pick it forever.

release_lvol_ns_slot is the rollback half: record + mini removed in one
transaction, releasing the slot atomically. All create/clone rollback
paths and the delete-path record removals go through it.

Also: get_next_available_subsystem_on_node treats only all_lvols=None as
"not provided" — an empty in-transaction snapshot result used to trigger
a fresh full-table read outside the caller's transaction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Commit 5c4c2ff gated every migration target-lvol create on
_ensure_lvstore_primary_leader, which queries bdev_lvol_get_lvstores —
a method the migration-tier mock SPDK server never implemented. Every
migration integration test has failed at start_migration since
("Lvstore lvs_tgt not found on <node>", 112 failures, red since
2026-07-27's first post-merge run).

The mock node is by construction the sole primary/leader of its
lvstore, so the handler reports lvs_primary + "lvs leadership" true and
returns an empty list for unknown lvs names (which the guard maps to
its not-found error). Registered in _METHOD_ERROR_CODES so failure-rate
injection exercises the guard's retry path too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

#1230)

_delete_replica_on_peer called rpc_client.subsystem_list(nqn), but
subsystem_list(self) takes no extra arguments at all -- it lists every
subsystem, unfiltered. Every call raised TypeError, silently swallowed by
the surrounding best-effort try/except and logged as a warning, so the
hublvol subsystem on a peer was never actually torn down during node
removal. Predates failure domains entirely -- introduced with node-removal
itself (#1104), found live while investigating an unrelated FD/1+1 removal
test (2026-08-10).

subsystem_get(nqn) already exists and does exactly what was intended here:
server-side filtered lookup, returning the matching subsystem dict or None.

…delete

Run mass_create_delete_docker-20260807-075600, LVS_1 (~7.5k objects):

1. The data-plane delete took 202s on average (max 675s, median 274s) while
   every RPC in it costs ~17ms. The API only issues the leader-side async
   delete; the sync legs on the non-leaders were left to lvol_monitor's
   serial loop, which drained 72 objects/min against a submit rate of
   153/min (peak 948/min) and built a ~2200-object backlog. Whenever that
   backlog was empty the same monitor finished a delete 0.1-0.2s after the
   async leg, so the cost was queueing, not per-object work.

   _delete_lvol_from_all_nodes now polls the leader's async delete to
   completion and issues the sync legs itself, the protocol
   _rollback_snapshot_bdev has always run inline (measured in that run at
   0.28s end to end for SNAP_1635). The poll is bounded (2s, vs the
   rollback path's 15s) and runs inside the leader's lvstore lock so the
   delete window stays exclusive. It fails CLOSED: on timeout, -35, ret 4,
   an RPC error or a non-int result no sync leg is issued and no durable
   task is queued -- the lvol stays in_deletion and lvol_monitor owns it
   exactly as before. Nodes finished inline are recorded in
   LVol.sync_deleted_nodes (atomic_update) and skipped by the monitor, so
   no node ever receives two sync deletes.

2. The monitors issued a sync delete on the LEADER as well. Its async
   delete has already removed the blob and unregistered the bdev, so the
   second pass re-walks the snapshot/clone metadata and errors on every
   entry the first pass cleaned: 4361 "Clone entry not found" plus 888
   lvol_delete_async_cb *ERROR* on the leader, and exactly 0 on either
   non-leader. Same signature as run 20260716 (1382x); the
   _remove_bdev_stack "already deleted, skipping" guard did not fire once,
   so it cannot be relied on to suppress it. Removed from both
   lvol_monitor and snapshot_monitor, matching the protocol stated in
   _rollback_snapshot_bdev: sync deletes go to every non-leader
   "unconditionally, never on the leader".

   Verified safe against bdev leaks: of the 162 objects that took the
   create-rollback path in that run (async on the leader, no leader sync
   leg), 0 were still present in the leader's end-of-run bdev dump.

process_lvol_delete_finish now re-reads the lvol record, because the copy
it receives comes from the cycle-start snapshot and would otherwise miss a
sync_deleted_nodes marker written by an in-flight API delete.

Snapshots keep the deferred sync stage (snapshot_monitor); only the
redundant leader leg is gone there. Unit tier passes (1218), ruff and mypy
clean; the integration tier needs FoundationDB + Docker and was not run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…or-name lookup

Cross-cluster cutover passed the transferhub CONTROLLER name
("<lvstore>/transferhub") to bdev_lvol_transfer_final_step, but only the
attached NAMESPACE bdev ("<lvstore>/transferhubn1") exists as a bdev. Every
cutover therefore failed with ENODEV (-19) and left the volume stuck in
cutover_pending, while snapshot replication kept working because it passes
the n1 bdev. ensure_hub_attached's second return value is now used, matching
tasks_runner_lvol_migration, and the gateway is logged.

Pool lookups that document "ID or name" had the resolve logic copied per
call site and the copies had drifted (delete_pool resolved by ID only, so a
valid name raised KeyError; add_replication --target-pool likewise, and it
stored the mutable name rather than the UUID). Consolidated into
DBController.get_pool_by_id_or_name and used from the drifted call sites.
add_replication also translates the KeyError from get_cluster_by_id into a
ValueError instead of surfacing a bare traceback to the operator.

Plus lab-script repairs for the 2-cluster async-replication harness
(setup_repl_test_2clusters, test_async_replication, collect_logs).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…t crash on inconclusive data-nic ping

During the 2026-08-10 helm-to-operator upgrade test, installing the R26
operator (Step 6) started the auto-restart task runner while 6 adopted
storage nodes were still OFFLINE with auto_restart_disabled=false and
not yet CR-linked (cr_namespace patched in a later step). It
auto-created 6 node_restart tasks whose health check built a k8s-service
hostname from the empty cr_namespace, producing an unresolvable
"..svc.cluster.local" name. The resulting inconclusive (None) ping
result then crashed task processing via an unguarded `bool |= None`,
wedging the tasks permanently active and blocking the later manual
`sn restart` with "Restart task found, can not restart storage node".

Fix two things:
- add_node_to_auto_restart now refuses to queue a restart for a
  kubernetes-mode node that hasn't been CR-linked yet, closing the
  window entirely instead of letting the task get created and fail.
  Docker deployments are unaffected (gated on cluster.mode).
- task_runner_node no longer crashes when a data-nic ping is
  inconclusive (None): only an explicit True flips the check, matching
  the tri-state handling already used elsewhere (tasks_runner_port_allow,
  storage_node_monitor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Depending on which path was taken the test would consistently fail. The
scenarios (source, target, unrelated) are all relevant and need to be
tested reliably. This PR adapts the tests s.t. one test case is
dedicated to each of the different outage variants.

This is not a supported use case and is not necessary to test.

This allows clients to distinguish different behaviors and avoids them
using bare-excepts.

K8sNativeResilientFailoverTest iteration 28 (2026-08-09) lost all I/O on
volume 638be965: ext4 went read-only with "no available path - failing
I/O". The volume had been running at 2 of 3 paths for the 11 minutes
before the outage, and the outage removed the two nodes holding those two
paths. The CSI node plugin had been reporting it the whole time
("Degraded subsystem active=2 expected=3") and could not repair it.

A path can go missing without anything looking wrong: nvme connect
succeeds, the target establishes qpairs, the client prints "new ctrl", no
keep-alive timeout and no controller reset ever fire -- but the namespace
never joins the multipath head, so the path does not exist for I/O
routing. On one node in that run, 71 lvol subsystems had a listener and
only 52 had a namespace.

Namespace: never publish a listener in front of an empty subsystem.
recreate_lvol_on_node logged a failed nvmf_subsystem_add_ns and fell
through to create the listener anyway, returning success. It now fails.
add_lvol_thread now checks the add_ns result and re-verifies namespace
presence before adding a listener, which also covers the second route
into this state -- an idempotency check that wrongly reports the
namespace already present (observed for 4 of the 19 orphans).

Controller ids: give every path a disjoint cntlid window. The host
rejects a controller presenting an already-seen cntlid ("Duplicate cntlid
N with nvmeX, subsys ..., rejecting") and that path is then gone for the
life of the connection. Two formulas existed -- 1000*(idx+1) on create,
1+1000*idx on recreate -- so the same node could get different windows
depending on which flow built its subsystem, and the path index silently
fell back to 0 for an unknown node, i.e. into the primary's window. Now
one helper, lvol_min_cntlid(), used by all call sites, with a fallback
above every assigned window instead of 0.

ANA promotion: fire once per offline episode, not once per cycle. The
monitor re-ran trigger_ana_failover_for_node on every pass for as long as
a node stayed offline: 2789 set_ana_state RPCs in 16 minutes at a flat
170/min. Each one is a real spdk_nvmf_subsystem_pause of a live subsystem
-- the RPC pauses before reaching SPDK's "state already matches"
short-circuit, and nvmf_subsystem_get_listeners pauses too, so neither
passing anagrpid nor reading the state first avoids it. Not making the
call is the only remedy.

Detection: re-enable the lvol monitor's subsystem sweep by default. It is
the only thing that sees a replica whose subsystem carries no namespace.
It was off for cost and for a mass-delete race; the race is now guarded
by the in_deletion re-reads, and the cost is bounded by a new
LVOL_MONITOR_SUBSYS_CHECK_INTERVAL_SEC (300s) evaluated once per cycle.
Adds a greppable "DEGRADED PATHS:" error naming the volume and replica,
so this state is visible above the client for the first time.

Not addressed here: the CSI-side repair loop cannot fix an
already-connected-but-pathless controller (nvme connect returns "already
connected"); that fix belongs in simplyblock-operator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

migration controllers merged under subssytem controller

…ackup fixes, cross cluster restores (#1208)

* Capture kubectl describe pod for stuck/timed-out FIO pods

When FIO pods are stuck in PodInitializing or time out during
wait_fio, save kubectl describe output to stuck_pod_describes/
directory for post-mortem debugging of volume mount or CSI failures.

* Fix checksum pod race and stale VolumeSnapshot between tests

TestSingleNodeOutage: The checksum utility pod was deleted with
--wait=false, causing a name collision when re-created 24s later.
The stale pod returned empty checksums, failing the assertion.
Fix: delete_pod() now accepts wait=True; _generate_checksums_dual
uses it to block until the pod is fully removed.

TestSingleNodeFailure: Stale VolumeSnapshots (snapshot-1, snapshot-2)
from TestSingleNodeOutage persisted because cleanup used --wait=false
and cleanup_k8s_leftovers only matched snap-* prefix. The next test's
kubectl apply hit "persistentVolumeClaimName is immutable", silently
reusing the stale snapshot that pointed to a deleted backend object.

Fixes:
- delete_pod/delete_volume_snapshot: add wait parameter
- _generate_checksums_dual: wait for pod deletion
- Teardown: wait for snapshot deletion + catch-all for untracked snapshots
- cleanup_k8s_leftovers: match snapshot-* in addition to snap-*
- create_volume_snapshot: detect and remove stale snapshot before apply

* Improve API parity audit: fix pool ordering, severity, and diagnostics

- Create audit pool in Phase 0 before read-only audits so pool.get,
  pool.iostats, volume.crud, and snapshot.crud are no longer skipped
  with "no pools available"
- Change all not_tested and interface_error severity from WARNING/INFO
  to ERROR — if an interface fails, it should fail the test
- Capture full API call details in every finding: the CLI command or
  HTTP method+path, HTTP status code, and response preview
- _run_cli now returns a dict with data, stdout, stderr, and command
  so failures include the actual stderr output for debugging
- Count mismatches now include sample IDs from each interface so
  it's clear which items are present vs missing
- Report HTML updated with new columns: API Call, HTTP Status,
  Response preview, and sample IDs for count mismatches
- pool.crud uses a separate pool name (parity_crud_pool) so it
  doesn't conflict with the audit pool lifecycle

* Add complete K8s upgrade operations guide (R25 Helm to R26+ Operator)

Full end-to-end runbook covering Phase 1 (R25.x legacy Helm deployment),
Phase 2 (pre-upgrade data setup with FIO/MD5/snapshots/clones), Phase 3
(10-step maintenance window migration), and Phase 4 (post-upgrade
validation including old data verify, new provisioning, and outage tests).

* Fix backup test bugs and exclude topology tests from backup pipeline

- TC-BCK-018: wait for backup completion before PVC deletion (K8s
  operator re-resolves PVC during reconciliation, causing
  BackupSourceResolutionError when PVC is deleted mid-backup)
- TC-BCK-172: pass restore_size="10G" for resized lvol restore (PVC
  size must match backup size)
- TC-BCK-175: remove -d debug flag that caused false positive error
  assertion on stderr
- Move topology backup tests (TestBackupAfterNodeAdd,
  TestBackupWithFioOnNewNode, TestBackupAfterNodeMigration,
  TestBackupDuringMigration) into separate get_backup_topology_tests()
  so they don't run in the regular backup pipeline without required
  NEW_NODE_IPS / migrate_to_worker params
- Add "backup-topology" keyword to e2e.py test runner

* Fix new_worker_nodes description: at least 1, not 2

Only TestSequentialNodeAdd needs 2 new nodes; all other add-node
tests work with 1.

* Align topology suite defaults with k8s-native workflows

- csi_repository: add default simplyblock/spdkcsi (was empty)
- csi_tag: add default latest (was empty)
- ifc_names: br-ex:enp2s0f0 (was ens18:enp1s0)
- cluster_environment: openshift-baremetal (was local)

* Add backup config support to K8s topology workflows

- Add cluster_security input (none/backup) to add-node and migration
  workflows (both workflow_call and workflow_dispatch)
- Deploy MinIO + backup-credentials secret when backup is enabled
- Add BACKUP_SPEC to StorageCluster CR for backup-enabled runs
- Auto-detect backup tests by testname containing "Backup"
- Pass cluster_security through topology suite parent workflows

* Fix 25-input limit for workflow_dispatch in migration workflows

Remove cluster_security from workflow_dispatch inputs (exceeds GitHub's
25-input limit). Keep it in workflow_call for programmatic use. Backup
is auto-enabled when testname contains "Backup" so no functionality lost.

* Fix TLS propagation: use truthy check instead of string comparison

Change `inputs.tls_enabled == 'true'` to `inputs.tls_enabled` in if
conditions. The truthy check works for both boolean true (from
workflow_call) and string "true" (from workflow_dispatch), avoiding
type coercion issues when parent workflow passes boolean to child.

* Don't label new_worker_nodes at bootstrap time

The DaemonSet schedules storage-node pods on any node with the
simplyblock.io/role=mgmt-plane label. Pre-labeling new_worker_nodes
caused pods to start on nodes not yet in the StorageNodeSet, resulting
in Init:CrashLoopBackOff. The test itself handles labeling when it
adds the node via StorageNodeSet CR update.

* Remove simplyblock label from new_worker_nodes during cleanup

Labels persist across pipeline runs. During cleanup, remove
simplyblock.io/role from new_worker_nodes and reset their hugepages
so the DaemonSet doesn't schedule storage-node pods on them before
the test adds them via StorageNodeSet CR.

* Clean stale /etc/simplyblock config, add sequential node expansion, delete stuck pods

Three fixes for K8s add-node and migration pipeline failures:

1. Pipeline cleanup: remove /etc/simplyblock from all worker nodes (both
   initial and new) during cleanup phase, preventing stale device config
   from causing init container CrashLoopBackOff on subsequent runs.

2. Add-node test: add new workers one at a time instead of all at once.
   Each node's StorageNode CR is created, stale pods are deleted, and the
   node is waited on to come online before proceeding to the next.

3. Both tests: after creating StorageNode/StorageNodeOps CRs, delete any
   existing simplyblock-storage-node-ds pods on the target worker so the
   DaemonSet recreates them with correct StorageNodeSet configuration.

* Pass driveSizeRange and pcieModel from StorageNodeSet to StorageNode CR

When creating StorageNode CRs for add-node expansion, read driveSizeRange
and pcieModel from the parent StorageNodeSet and include them in the
overrides block. This ensures the init container can discover the correct
SSD devices on the new worker node.

* Wait for per-node-config ConfigMap before deleting stale pods

Root cause: after creating a StorageNode CR, the test immediately deleted the
stale DaemonSet pod. The operator hadn't yet updated the per-node-config
ConfigMap with the new worker's MAX_LVOL value, so the recreated pod started
with MAX_LVOL=0 and crashed in s-node-api-config-generator init container.

Fix: poll the per-node-config ConfigMap until it has an entry for the worker
node before deleting any stale pods. This ensures the DaemonSet recreates the
pod with the correct configuration.

* Fix topology suite summary parsing and stale pod handling

Three issues fixed:

1. TestSequentialNodeAdd / TestAddNodeSnapshotCloneOnNewNode fail with
   "Only 4/5 snode-spdk pods" because they don't wait for the operator to
   populate the per-node-config ConfigMap before the DaemonSet pod starts.
   Added wait_for_per_node_config + delete_storage_node_pods_on_worker
   calls to both K8s code paths in test_add_node_edge_cases.py.

2. Topology suite Slack summaries show "?/? passed, ? failed" because the
   regex patterns only handle the k8s-native summary format (inline
   "**Total:** N") but not the e2e-bootstrap format (table with emojis).
   Updated all three parent workflows to handle both formats.

3. K8s child workflows always send individual Slack notifications even when
   send_slack_notification=false because the condition
   (inputs.send_slack_notification || 'true') == 'true' evaluates to true
   for both true and false inputs. Changed to != false.

* Move ConfigMap wait and stale pod deletion before StorageNodeOps creation

The wait_for_per_node_config and delete_storage_node_pods_on_worker calls
were placed AFTER the StorageNodeOps CR creation, which meant our test
was deleting pods while the operator was actively managing the migration.
This broke the operator's DNS/endpoint resolution, causing it to hang at
"waiting for DNS to be published" indefinitely.

Move these calls BEFORE the StorageNodeOps CR creation so the stale
crashing pod (MAX_LVOL=0) is fixed first, and the operator finds a
healthy pod when it starts the migration.

* Add MinIO trace logging to backup test workflows, align upgrade pipeline with UPGRADE.md

- Add mc admin trace background logging to 8 workflow files for backup test runs:
  K8s (port-forward): k8s-native-e2e, k8s-native-e2e-node-migration, k8s-native-e2e-add-node
  Docker (direct): e2e-bootstrap, stress-run-bootstrap, monitoring-suite-docker,
  upgrade-bootstrap, upgrade-bootstrap-single
- Trace logs are saved as artifacts for post-run debugging
- Align k8s_major_upgrade.py with UPGRADE.md: add post-upgrade old data verification
  (FIO verify-only, fresh IO, new snapshots on old PVCs), node outage test, pre-upgrade
  state capture, and final checklist assertions
- Update UPGRADE.md: add worker node labeling section, Pool CR name must match backend,
  StorageCluster CR name must match upgrade secret
- Add worker node label step to k8s-native-upgrade.yaml for R25 storage plane discovery

* Remove stale storagenodeset labels during cleanup in all K8s workflows

Previous test runs leave io.simplyblock.storagenodeset labels on worker
nodes.  When a new run starts, the operator re-deploys the DaemonSet
which immediately schedules pods on all labeled nodes — including
migration targets and add-node spares that aren't in the StorageNodeSet
workerNodes list.  Those pods crash with MAX_LVOL=0 because the
per-node-config ConfigMap has no entry for them.

Add a cleanup step to all five K8s workflows that removes the
storagenodeset label from all worker nodes before the new run begins.

* Fix TARGET_DOCKER_IMAGE default from main-latest to main

The simplyblock image tag is 'main', not 'main-latest'. The incorrect
default caused upgrade tests to fail when no custom image was specified.

* Add target worker to StorageNodeSet before migration

The operator expects a Running storage-node pod on the migration target
worker before it can process a StorageNodeOps CR. Previously, the pod
only existed due to stale labels from earlier runs. On a clean cluster,
no pod was scheduled and the operator hung at "waiting for storage-node
pod on worker".

Fix by adding the target worker to the StorageNodeSet (via StorageNode
CR with expand=true) before creating the StorageNodeOps. This follows
the same pattern as add-node: create CR -> wait for ConfigMap -> fix
stale pods -> wait for snode-spdk pod -> then proceed with migration.

* Revert "Add target worker to StorageNodeSet before migration"

This reverts commit 439a0d0.

* Fix MinIO trace setup: use secret keys and resilient mc install

- Replace hardcoded minioadmin credentials with MINIO_ACCESS_KEY /
  MINIO_SECRET_KEY env vars from GitHub secrets in all 5 Docker
  workflows (e2e-bootstrap, monitoring-suite-docker, upgrade-bootstrap,
  upgrade-bootstrap-single, stress-run-bootstrap)
- Add /tmp fallback for mc binary install when /usr/local/bin write
  fails (curl exit 23 on runners with permission/disk issues)
- Fix TEST_CLASS defaults: use exact class names (TestMajorUpgrade,
  TestMajorUpgradeSingleNode) instead of substrings that match
  multiple test classes

* Add preserve_resources_on_failure to K8s topology pipelines

The node-migration and add-node K8s pipelines were missing this input,
so test teardown deleted lvols while FIO was still running, causing
spurious err=121 (Remote I/O error). Default to true (matching k8s-e2e).

* Fix 25-input limit for node-migration workflow_dispatch

Move preserve_resources_on_failure to workflow_call only in the
migration pipeline and topology suite (same pattern as cluster_security).
Defaults to true when not provided.

* Add cleanup-simplyblock.sh to all K8s workflow cleanup steps

Run the operator's cleanup-simplyblock.sh before cleanup_k8s.sh for
more thorough cleanup of stale resources after failed migrations.
Also fixes 25-input limit for node-migration workflow_dispatch by
moving preserve_resources_on_failure to workflow_call only.

* Remove hardcoded pcieModel from K8s StorageNodeSet specs

Not needed for openshift-baremetal and openshift lab clusters.
Removed from all 7 K8s workflow files (8 occurrences total).

* Make pcieModel conditional: skip for openshift-local and openshift-baremetal

pcieModel is not needed on openshift-local and openshift-baremetal clusters.
The PCIE_MODEL_YAML variable is now set conditionally based on
cluster_environment, matching the existing RESERVED_CPU_YAML pattern.

* Stop upgrade test runner after first test failure

When an upgrade test fails, the cluster state is unknown and subsequent
tests will fail too (as seen with TestMajorUpgradeSingleNode failing
because the node was still offline from TestMajorUpgrade). Add
stop_after_teardown flag to break the test loop after collecting logs
and management details for the failed test.

* Pass --spdk-proxy-image on sn restart during upgrade tests

The upgrade test was only passing --spdk-image (new SPDK) but not
--spdk-proxy-image, so the restart used the old proxy image from the
node's DB record (26.2.8-PRE). This version mismatch (new SPDK + old
proxy) caused 157ms attach latency (vs normal 5-7ms) and contributed
to hublvol attach race condition failures.

Both Docker (major_upgrade.py) and K8s (k8s_major_upgrade.py) upgrade
tests now pass the target docker image as --spdk-proxy-image.

* Add dual-node-per-host test classes and fix K8s upgrade workflow

- Add Docker dual-node tests: TestMajorUpgradeDualNode, TestAddNodesDualNodePerHost
- Add K8s dual-node tests: K8sNativeMajorUpgradeDualNode, TestAddK8sNodesDualNodePerHost
- Add nodes_per_socket param to deploy_storage_node() in ssh_utils
- Register new test classes in __init__.py and add guards in e2e.py
- Fix k8s-native-upgrade.yaml: add KUBECONFIG setup step from secret
- Fix k8s-native-upgrade.yaml: gate cert-manager install on upgrade_type != r25-to-r2x

* Rename operator_repo_branches to helm_repo_branches in k8s upgrade workflow

* Fix KUBECONFIG setup: try existing kubeconfig before falling back to secret

Self-hosted OpenShift runners already have cluster access configured.
Only write from K8S_KUBECONFIG secret as a fallback.

* Use per-environment kubeconfig secrets matching other K8s native pipelines

Select kubeconfig secret based on cluster_environment input, matching the
pattern used by k8s-native-e2e, stress, add-node, and migration workflows.
Add kubeconfig cleanup step.

* Add helm dependency build before R25 chart installs

The sbcli control plane chart has dependencies (mongodb, opensearch,
prometheus, etc.) that must be fetched before install.

* Add old deployment cleanup to k8s upgrade workflow

Matches cleanup pattern from k8s-native-e2e and other K8s native
pipelines: uninstall old Helm releases, delete CRDs/finalizers, reset
hugepages, and clean cert-manager before fresh bootstrap.

* Add set -euxo pipefail to all K8s helm install steps for debug output

Prints exact helm commands with all resolved values in CI logs so
install issues can be diagnosed from the workflow output.

* Fix R25 admin pod detection: exclude ingress controller from grep

The pattern 'webappapi|admin|sbcli' incorrectly matched
sbcli-ingress-controller before simplyblock-admin-control.
Narrowed to 'admin-control|webappapi' to match only pods
that have sbcli-dev installed.

* Parallelize hugepages reset with 120s timeout to prevent cleanup hang

The oc debug calls for hugepages reset and kubelet restart were running
sequentially across 6 workers with no timeout, causing 25+ minute
cleanup times. Now runs all nodes in parallel with 2-minute timeout
per operation.

* Rewrite cleanup to force-delete everything with no graceful waits

Phases: uninstall helm releases (--no-hooks), strip all finalizers in
parallel, force-delete CRDs/namespace/PVs, reset hugepages in single
combined oc debug call per node, poll for namespace deletion with
continuous finalizer stripping. 10-minute hard timeout on entire step.

* Fix cleanup: strip PV/PVC finalizers before namespace delete, reduce wait spam

- Phase 4: check namespace exists before stripping resource finalizers
- Phase 5: strip PV finalizers + claimRef before force-deleting PVs
- Phase 8: reduce wait iterations, only strip finalizers if namespace exists

* Disable ingress-nginx admission webhook for R25 sbcli chart install

The webhook ValidatingWebhookConfiguration gets registered before the
ingress controller pod is ready, causing intermittent "no endpoints
available" failures during helm install.

* Fix cleanup Phase 8: use namespace finalize API instead of api-resources loop

The api-resources loop iterated over ALL namespaced resource types
including events, which keep getting recreated and caused the cleanup
to loop endlessly. Now uses kubectl replace --raw to strip namespace
finalizers directly, and only patches a fixed list of blocking resource
types (skipping events, endpoints, etc.).

* Fix cluster secret parsing: use 'cluster get-secret' instead of awk on table

The cluster list table output doesn't contain the secret column, so
awk $NF was grabbing the pipe '|' character. Now uses the dedicated
'cluster get-secret <id>' command with JSON fallback.

* Fix R25 spdk-csi chart path: v0.2.4 uses charts/ not csi-driver/charts/

The simplyblock-operator v0.2.4 tag has the spdk-csi chart at
charts/spdk-csi/latest/spdk-csi/ while main has it under
csi-driver/charts/. Added path detection to support both layouts.

* Improve cleanup: force-delete resources in Phase 4, finalize namespace in Phase 6, increase timeout to 15m

- Phase 4 now force-deletes resources (not just strips finalizers),
  so pods/deployments don't linger in Terminating state
- Phase 5 splits PV patches to avoid invalid JSON merge
- Phase 6 immediately strips namespace finalizers via API after delete
- Phase 7 increased per-node timeout to 90s for oc debug
- Phase 8 simplified to just verify + finalize (no resource iteration)
- Overall timeout increased from 10 to 15 minutes

* Set storagenode.coresPercentage=50 for R25 spdk-csi chart install

SPDK pods fail to schedule with 'Insufficient cpu' on baremetal
workers when using the default coresPercentage.

* Fix R25 pool creation: use direct CLI instead of Pool CRD

R25 has no operator to reconcile Pool CRDs, so add_storage_pool()
(which creates a Pool CRD and waits for reconciliation) times out
with "Pool not visible in sbcli after 300s".

Added add_storage_pool_direct() to K8sSbcliUtils which calls
'sbcli-dev pool add' via kubectl exec. The maintenance upgrade
path now uses this method with sbcli_cmd="sbcli-dev".

* Pass cluster params to R25 cluster create and add cluster activate

R25 cluster create was missing --ndcs/--npcs/--bs/--chunk-bs/--jm-count
so clusters defaulted to ndcs=1,npcs=1. Also R25 has no auto-activate,
so added explicit 'cluster activate' after all storage nodes register.

* Delete stale StorageClasses and VolumeSnapshotClasses during cleanup

Previous test runs leave behind cluster-scoped StorageClass and
VolumeSnapshotClass objects with provisioner=csi.simplyblock.io.
These are not cleaned up by namespace deletion. Now deleted in
the cleanup phase by checking the provisioner/driver field.

* Fix R25 pre-upgrade: use chart-created StorageClass, align pool name, set storagenode ndcs/npcs

- R25 spdk-csi chart auto-creates StorageClass 'simplyblock-csi-sc'
  from logicalVolume config. Maintenance upgrade now uses it instead
  of creating its own (which would fail without an operator).
- Changed logicalVolume.pool_name from 'testing1' to 'testpool' to
  match the pool created by the test via sbcli-dev.
- Set storagenode.numDataChunks and numParityChunks from cluster
  params (were defaulting to 1).

* Fix R25 spdk-csi install and test to match actual R25 deployment flow

Workflow:
- logicalVolume.pool_name set to 'testing1' (matching R25 convention)
- Removed logicalVolume.snapshot (not in R25 chart)
- storagenode.numPartitions=0 (matching R25 default)
- Removed storagenode.numDataChunks/numParityChunks (not valid R25 params)
- Added --create-namespace to match actual R25 install command

Test (_run_maintenance_upgrade):
- Pool created as 'testing1' to match chart's logicalVolume.pool_name
- Skips _create_storage_classes() entirely — uses the chart-created
  'simplyblock-csi-sc' StorageClass from the logicalVolume config
- Maps XFS SC to the same chart SC (R25 has no XFS variant)

* Mask cluster secret in logs and set numPartitions=1

- Add ::add-mask:: before writing CLUSTER_SECRET to GITHUB_ENV at all 3 locations
- Wrap secret retrieval in set +x/set -x to prevent bash trace leaking secret
- Change storagenode.numPartitions from 0 to 1 for R25 spdk-csi install

* Redact cluster secret from pre-upgrade state log

* Remove unsupported cluster params from R25 cluster create

* Enable preserve_resources_on_failure by default in upgrade tests

* Add nvme disconnect-all to cleanup phase in all K8s workflows

Prevents stale NVMe-oF connections from causing nvme connect failures
in subsequent test runs (Invalid argument on /dev/nvme-fabrics).

* Make pre-upgrade FIO non-fatal in R25 maintenance upgrade test

The upgrade test should not fail if pre-upgrade FIO doesn't complete.
The goal is testing the upgrade path, not the old version's IO.

Changes:
- Reduce FIO runtime from 120s to 60s
- Wait up to 5 mins for FIO, catch failures as warnings
- Clean up FIO pods before taking snapshots
- Create snapshots/clones without running FIO on clones

* Add MD5 checksum verification and clone FIO to R25 maintenance upgrade

Pre-upgrade flow:
1. Create PVCs, run FIO (60s, non-fatal)
2. Clean up FIO pods
3. Create snapshots and clones (no FIO on clones initially)
4. Run FIO on clones (60s, non-fatal)
5. Capture MD5 checksums on all PVCs and clones

Post-upgrade:
- Verify MD5 checksums match pre-upgrade data
- Ensures data integrity survived the maintenance window

* Fix FIO cleanup to preserve PVCs/snapshots/clones for MD5 checksums

cleanup_stale_fio_resources() deletes clone PVCs, snapshots, and test
PVCs along with FIO jobs. Replace mid-test calls with a targeted
_cleanup_fio_jobs_only() that only removes FIO jobs and configmaps,
keeping PVCs available for utility pod mounting and md5sum.

* Use force flag on node shutdown during maintenance upgrade

Older versions require --force to shut down nodes that aren't in
suspended state. The suspend call may silently fail, leaving nodes
online and causing shutdown to error with "Node is not in suspended
state". Using force=True bypasses this check.

* Skip suspend, use shutdown --force directly in maintenance upgrade

Suspend fails with "Offline storage nodes found, cannot suspend node
without --force" when any node is already offline (Step 6.1 scenario).
Remove the suspend step entirely and just use shutdown --force which
bypasses all state checks.

* Wait for all SPDK pods ready and nodes online after sequential restart

After Step 10 restarts all nodes one at a time, add an explicit wait
for all SPDK pods to reach Ready state and all storage nodes to be
online before ending the maintenance window. Prevents proceeding to
post-upgrade steps while a node is still coming up.

* Wait for all storage nodes online before R25 cluster activate

After nodes register in sn list, poll until all report status=online
before calling cluster activate. Prevents activating with nodes still
starting up (SPDK pod NotReady / node offline).

* Install cert-manager in test before operator helm install

R25 clusters don't have cert-manager since TLS wasn't supported.
The target operator chart's validate-tls.yaml requires cert-manager
CRDs when tls.enabled=true. Install cert-manager inside the test's
_install_operator_chart if TLS is enabled and CRDs are missing.

* Add retry and stale cleanup to cert-manager install, revert helm retry

cert-manager _ensure_cert_manager now:
- Uninstalls stale cert-manager release before install
- Retries install up to 3 times, uninstalling between attempts

Reverted helm install retry/uninstall logic - not needed there.

* Add rapid-restart stress test: 6000 objects, 30x stop/restart cycles

New test classes MassCreateRapidRestart_6k_3Snap_Docker and
MassCreateRapidRestart_6k_3Snap_K8s: create 1500 lvols + 4500 snapshots
(1:3 ratio), run 30 container stop/restart cycles without waiting for
migration (60s cooldown), then delete lvols, create clones, and repeat
30 more restart cycles. Final summary prints per-iteration stop-to-online
times for both phases (60 entries total).

* Support NEW_NODE_IPS for cross-cluster restore auto-bootstrap

_bootstrap_second_cluster() now collects spare node IPs from both
STORAGE_PRIVATE_IPS and NEW_NODE_IPS env vars.  This aligns with the
existing e2e-bootstrap.yml workflow which cleans NEW_NODE_IPS hosts
without adding them to cluster 1 — making them ideal cluster 2 candidates.

Also add TestBackupCrossClusterRestore to TOPOLOGY_MODIFYING_TESTS so
inter-test cluster reset triggers when needed.

* Register MassCreateRapidRestart tests in stress test discovery

Add MassCreateRapidRestart_6k_3Snap_Docker and K8s to imports,
ALL_TESTS, get_stress_tests(), and get_monitoring_tests().

* Fix NS_PER_SUBSYSTEM exceeding 50 hard limit in rapid restart tests

The API enforces max_namespace_per_subsys=50. With NUM_SUBSYSTEMS=10,
entity cap reduced 1500 lvols to 150/subsystem which was rejected.
Changed to 30 subsystems x 50 ns/sub = 1500 lvols.

* Fix spdk-csi snapshot-controller blocking operator helm install

The old spdk-csi helm chart sets helm.sh/resource-policy: keep on the
simplyblock-snapshot-controller Deployment in kube-system.  This causes
the resource to survive helm uninstall, but it retains the stale
meta.helm.sh/release-name: spdk-csi annotation.  When the new
simplyblock-operator chart tries to create the same resource, helm
refuses with "invalid ownership metadata".

Replace the incorrect re-annotation approach (_readopt_spdk_csi_resources)
with explicit deletion of the orphaned resource after helm uninstall
spdk-csi, in _uninstall_helm_releases().  The new operator chart then
creates its own version cleanly.

* Enforce max 50 ns/subsystem API limit across all mass create tests

The API rejects max_namespace_per_subsys > 50. Instead of fixing each
test class individually, add enforcement in both orchestrator methods
(_run_mass_create_delete_test and _run_mass_create_rapid_restart_test)
that automatically redistributes lvols into more subsystems when
NS_PER_SUBSYSTEM exceeds MAX_NS_PER_SUBSYSTEM (50).

* Fix FDB cluster-config ConfigMap lost during helm uninstall sbcli

After helm uninstall sbcli, the simplyblock-fdb-cluster-config ConfigMap
is deleted despite resource-policy:keep annotations on other FDB
resources.  The new operator's admin-control pods mount this ConfigMap
and get stuck in ContainerCreating: "configmap simplyblock-fdb-cluster-
config not found".

Three fixes:
1. Add ConfigMap to _FDB_KEEP_RESOURCES so it gets annotated with
   resource-policy:keep before helm uninstall.
2. Capture the FDB cluster file data BEFORE helm uninstall, and
   recreate the ConfigMap if it's missing afterward (fallback for
   cases where keep annotation doesn't work, e.g. resource owned
   by a different sub-chart).
3. Add explicit wait for admin-control pods to reach Ready state
   after operator chart install, with diagnostic event logging if
   pods remain in ContainerCreating.

* Document new upgrade steps in UPGRADE.md

- Step 1: Add ConfigMap simplyblock-fdb-cluster-config to FDB keep resources
  (8th resource). Admin pods mount this as fdb-cluster-file volume.
- Step 2: Use shutdown --force instead of separate suspend+shutdown commands
- Step 3.1: Delete orphaned simplyblock-snapshot-controller deployment in
  kube-system after helm uninstall spdk-csi (resource-policy: keep causes it
  to survive with stale ownership annotations)
- Step 4.1: Verify FDB cluster-config ConfigMap survived helm uninstall sbcli,
  with recovery procedure to recreate from FDB pod if missing
- Step 6: Add cert-manager prerequisite for TLS-enabled installs

* Use shared cleanup scripts in k8s-native-upgrade workflow

Replace the inline cleanup logic with calls to cleanup-simplyblock.sh and
cleanup_k8s.sh, matching the pattern used by k8s-native-e2e.yaml. The inline
cleanup was missing kube-system resource deletion (snapshot-controller),
StorageClass cleanup, webhook cleanup, and other steps that the shared
scripts handle. This caused R25 spdk-csi helm install to fail with
"existing resource conflict" for simplyblock-snapshot-controller when the
deployment survived from a previous upgrade test run.

* Add R25 helm uninstall and kube-system cleanup to upgrade workflow

The previous cleanup only ran shared scripts designed for R26 operator
deployments. Add explicit helm uninstall for R25 charts (sbcli, spdk-csi)
and delete simplyblock resources in kube-system (snapshot-controller with
resource-policy: keep) before running the shared cleanup scripts. This
ensures both R25 and R26 leftovers are cleaned between test runs.

* Change default cluster_params in k8s-native-upgrade workflow

Update defaults from ndcs=2,npcs=2,partitions=1,jm_count=4 to
ndcs=1,npcs=1,partitions=0,jm_count=3,max_lvol=30. Also fix the
fallback value in the parse step to include max_lvol and match the
input default.

* Add FDB readiness wait and secret validation in upgrade workflow

The R25 setup was attempting cluster create before FDB was fully
initialized, causing sbcli-dev commands to return FDB error messages
(Connection string invalid 2104) instead of actual data. The error
text was captured as CLUSTER_SECRET and passed to helm install --set,
which failed with "key has no value" due to commas in the error.

Fix:
- Add FDB readiness loop (30 x 10s) before cluster create that checks
  for FDB error patterns in sbcli-dev output
- Validate CLUSTER_SECRET length (must be <= 100 chars) at all three
  capture points — fail early with a clear error instead of passing
  garbage to helm install

* Fix wrong label selector for admin-control pods in upgrade test

The wait loop in _install_operator_chart used the label
app.kubernetes.io/component=admin-control, but the operator chart
deploys admin pods with label app=simplyblock-admin-control. This
caused the wait loop to find nothing for 300s, logging a misleading
"Admin-control pods did not become Ready" error even though the pods
were actually Running and Ready.

* Add FDB CRD protection step and comprehensive FDB verification to UPGRADE.md

Root cause analysis from 2026-08-07 upgrade run: all FDB resources
disappeared after helm uninstall sbcli despite keep annotations. Most
likely cause: the sbcli chart includes FDB CRDs, and helm uninstall
deletes CRDs, which triggers Kubernetes cascade deletion of all CRs of
that type — bypassing helm.sh/resource-policy=keep entirely.

Add:
- Step 1.1: Protect FDB CRDs from helm deletion (needs dev confirmation)
- Step 4: Comprehensive FDB verification checklist (CR, deployment,
  pods, CRDs) with clear failure guidance

* Fix FDB keep annotations: patch Helm release secret instead of kubectl annotate

kubectl annotate on live resources does not protect against helm uninstall
because Helm reads annotations from its stored release manifest, not from
etcd. Updated the E2E test to decode/patch/re-encode the Helm release
secret so the keep annotation is in Helm's stored manifest. Also updated
UPGRADE.md with the correct approach and added R25 manual setup steps.

* Add comprehensive cleanup script for R25 + R26 upgrade test setups

Standalone script that cleans both R25 (sbcli/spdk-csi) and R26
(operator) setups so the next upgrade test run starts completely fresh.
Covers: Helm releases, kube-system leftovers, CRs, snapshots, PVCs,
PVs, CRDs, cert-manager, namespaces, NVMe disconnect, hugepages reset,
kubelet restart, node labels, and CSI hostpath data.

* Fix FDB keep annotations: patch Helm release secret instead of kubectl annotate

kubectl annotate on live resources does not protect against helm uninstall
because Helm reads annotations from its stored release manifest, not from
etcd.

Primary approach: edit R25 chart template files on disk to add
helm.sh/resource-policy: keep annotations, then run helm upgrade
--reuse-values to persist them into Helm's stored release manifest.

Fallback: decode/patch/re-encode the Helm release secret directly
if the chart path is not available.

Also passes R25_CHART_PATH env var from workflow to E2E test, and
disables the API parity scheduled run to avoid interfering with
other pipeline runs.

* Change default SBCLI_BRANCH from R25.10-Hotfix to main in e2e and stress pipelines

* Add Graylog/OpenSearch log collection to Docker upgrade pipelines

The upgrade pipelines (multi-node, single-node, single-node-v2) had a
placeholder note about Graylog logs but never actually collected them.
This adds the same collection step used by e2e-docker.yml: SSHes to
mgmt node, runs collect_logs.py in 1-hour chunks (newest-first), probes
OpenSearch then falls back to Graylog, with adaptive retry (60m→5m→1m),
and saves results to ${RUN_BASE_DIR}/graylog_collected/.

* Add defensive cancel-task and disable-auto-restart methods for upgrade

Add _cancel_stale_restart_tasks() and _disable_auto_restart_all_nodes()
to handle stale node_restart tasks that block sn restart during upgrade.
Both calls are commented out since dev fixed the operator-side auto-restart
issue, but kept as safety nets for potential regressions.

Also documents Step 2.1 in UPGRADE.md for manual upgrade procedures.

* Fix maintenance upgrade: don't wait for cluster active between node restarts

In the maintenance upgrade path all nodes start offline. Waiting for
cluster active after restarting the first node blocks forever because the
cluster needs all nodes online. Removed per-node cluster-active and
migration checks from _restart_nodes_sequentially — the caller already
waits for cluster active after all nodes are restarted.

* Fix dual-node maintenance upgrade: same cluster-active wait issue

The dual-node override of _restart_nodes_sequentially also waited for
cluster active between workers, which blocks when all nodes start
offline. Removed per-worker cluster-active and migration checks — the
caller handles these after all nodes are restarted.

* Fix post-upgrade health check: retry until health_check settles to True

After a maintenance upgrade all nodes restart nearly simultaneously.
The health_check field transitions None → False → True as the
monitoring loop catches up, which can take 20-30 seconds.  The
previous assertion checked once and failed immediately if any node
had health_check=False.

Replace the one-shot assert with a polling loop (120s timeout,
10s interval) that waits for all nodes to report health_check=True
before declaring failure.

* Add cross-cluster restore CI pipeline and remove old k8s-e2e workflow

- Add cross-cluster-restore.yml: Docker-based pipeline for testing
  backup/restore across two clusters on the same mgmt node. Takes
  storage node IPs, validates min 2*(NDCS+NPCS) nodes, splits evenly
  between cluster-1 and cluster-2, then calls e2e-bootstrap.yml with
  TestBackupCrossClusterRestore which auto-bootstraps the second cluster
  from spare nodes.
- Remove k8s-e2e.yaml: Legacy AWS/Terraform-based K8s E2E pipeline,
  superseded by newer on-prem workflows.

* Add K8s native cross-cluster restore pipeline and test support

Pipeline (.github/workflows/k8s-native-cross-cluster-restore.yaml):
- Deploys two simplyblock clusters in separate K8s namespaces
  (simplyblock + simplyblock-c2) on the same K8s cluster
- Splits worker nodes evenly between clusters (validates min 2*(NDCS+NPCS))
- Shared MinIO in minio namespace with backup credentials in both namespaces
- Two Helm installs + two StorageCluster/Pool/StorageNodeSet CRD sets
- Waits for both clusters active, extracts credentials for both
- Runs TestBackupCrossClusterRestore with --run_k8s True and CLUSTER2_* env vars

Test (e2e/e2e_tests/backup/test_backup_restore.py):
- Remove K8s mode skip from TestBackupCrossClusterRestore.run()
- Add _init_k8s_c2(): creates second K8sUtils for C2 namespace
- Add _discover_k8s_cluster2(): extracts C2 cluster ID/secret from
  admin pod in C2 namespace when CLUSTER2_* env vars not pre-set
- Update _sbcli_c2() to route commands through kubectl exec into C2
  namespace admin pod in K8s mode
- Update _export_backup_metadata() to transfer metadata file between
  C1 and C2 admin pods via kubectl cp in K8s mode

* Fix ruff lint errors in test_backup_restore.py

- Remove unused type annotation on _k8s_c2 (F821: undefined name K8sUtils)
- Remove unused mgmt_ip variable in _remove_nodes_from_cluster1 (F841)

* Fix ruff lint errors in k8s_major_upgrade.py and mass_create_delete_stress.py

- Remove unused top-level `import re` (local import exists at usage site)
- Remove unused `pre_upgrade_fio_ok` variable
- Remove unused `restart_ts` variable
- Remove unused `max_dur` and `test_start` variables
- Rename ambiguous loop variable `l` to `ln` (E741)

* Fix admin-pod recycling crash and preserve resources on failure

Three changes to stabilize the K8s native upgrade test:

1. exec_sbcli: Detect "pod does not exist" and "pod not found" errors
   (not just "NotFound") when the admin-control pod is recycled by the
   R26 operator during node restarts.  Re-resolve the pod and retry.

2. wait_for_storage_node_status: Catch transient JSONDecodeError /
   IndexError from get_storage_node_details instead of crashing the
   polling loop.  The admin pod may be briefly unavailable during
   operator reconciliation.

3. upgrade_e2e.py: Respect preserve_resources_on_failure — skip K8s
   resource cleanup (PVCs, lvols, pools) when a test fails, matching
   the behavior already present in e2e.py and stress.py.

Also disable K8sNativeMajorUpgradeDualNode from the upgrade test list
to focus on single-node-per-host upgrade first.

* Add K8s-native CRD flow for cross-cluster restore and remove e2e-bootstrap-k8s pipeline

Replace CLI-based cross-cluster restore with K8s-native CRD flow
(StorageBackup → BackupImport → BackupRestore) when running in K8s mode.
The controller handles source-switching automatically. Docker/CLI mode
is preserved as a separate code path. Also removes the deprecated
e2e-bootstrap-k8s.yml workflow.

* Update UPGRADE.md with lessons learned from E2E upgrade runs

Key changes:
- Step 2.1: Note product-side auto-restart fix, mark as safety net
- Step 9.1: Add cancel-task step for stale node_restart tasks
- Step 10: Fix incorrect guidance — do NOT wait for cluster active
  between individual node restarts in maintenance path (cluster stays
  suspended until all nodes are online)
- Step 10.1: New step — wait for cluster active and health_check to
  settle (120s timeout) after all nodes are restarted
- Add operational notes section covering admin-pod recycling,
  health_check settling delay, StorageNodeSet CR adoption, and
  preserve-resources-on-failure for debugging

A cross-cluster DR fail-over produced five volumes that read as ALL ZEROS:
no filesystem, blkid empty, `mount -t xfs` failing with a bad superblock, md5
mismatch on every volume. The fail-over itself reported success — all five
replication objects reached failed_over and valid connection strings were
returned — so the loss was silent.

_last_replicated_target_snapshot accepted any snapshot whose
target_replicated_snap_uuid was merely SET. That field is populated when the
target copy is ALLOCATED, not when its data arrives, and the selector never
looked at the replication task's status: with five snapshot_replication tasks
still `running`, an in-flight transfer was an eligible fail-over point. It also
never checked that the chosen target copy still existed — in the failing run all
five `cloned_from` parents were gone from the snapshot table while dozens of
replicated snapshots sat in in_deletion, so each clone was left with an orphaned
parent.

Require the replication task to be STATUS_DONE, skip a target copy that is
missing or in_deletion, and walk candidates newest-first so a bad newest
snapshot falls back to the last good one. Return None rather than hand back a
volume full of zeros.

This hardens which snapshot a fail-over selects. If retention is genuinely
deleting a live clone's parent, that is a separate bug and still open.

Also:
- collect_logs.py sourced everything from Graylog with no fallback, so with
  Graylog not ingesting it wrote 0-byte logs and still reported "Done!" with a
  plausible-looking tarball — the investigation above had no service logs at all
  for the failing window. Warn loudly on zero control-plane lines and, in docker
  mode, fall back to `docker service logs`, which spans task restarts (plain
  `docker logs` only holds the live task, so a service recreated mid-incident
  loses its earlier output).
- test_async_replication.py: bound cleanup on both sides (`timeout 15` per
  umount plus a per-command SSH timeout — a hung umount used to stall 15 min per
  mount), kill leftover nohup fio before unmounting, and let read-phase SSH
  failures on best-effort commands continue instead of killing a passing run.
- test_failover_target.py: its task fixture left JobSchedule.status at "" while
  asserting a successful fail-over; set STATUS_DONE to match the intent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A hublvol controller's paths span two axes at once: the LVS leader's data
NICs (ANA optimized, storage_node.py:399) and the failover node's (ANA
non_optimized, storage_node.py:518). Those axes want opposite policies —
round-robin across the leader's NICs, strict standby for the failover node.

hublvol_reconnect.py attaches with multipath="multipath" but never set a
multipath policy: the only bdev_nvme_set_multipath_policy call in the tree
was the remote-device/JM one in storage_node_ops._connect_device. So every
hublvol bdev kept SPDK's creation default ACTIVE_PASSIVE
(bdev_nvme.c:4690) and one NIC of the leader carried all hub IO, leaving
the second path unexercised until failover discovered it. That is how a
tertiary ends up effectively single-pathed to its hub, which the
2026-08-03 six-node cascade turned on (4426's tertiary was down to one hub
path, pointing at the node that then self-evicted).

Asserting active_active gets both axes right at once, because SPDK
load-balances only WITHIN an ANA state: _bdev_nvme_find_io_path
(bdev_nvme.c:1150) returns the first available optimized path in its
round-robin scan and reaches non_optimized only when no optimized path
exists. The leader/failover preference therefore stays expressed purely in
ANA and is not weakened. Selector is left at SPDK's active_active default
(round_robin, rr_min_io coerced UINT32_MAX -> 1 at bdev_nvme.c:5626),
matching the remote-device path.

- ensure_hublvol_active_active(): idempotent assertion on <ctrl>n1, with a
  bounded poll for the AER-driven namespace bdev to surface. wait=False
  for callers inside the LVS-rejoin freeze / port-block window so the
  sub-second budget absorbs no retry loop.
- Called from _reconcile_under_lock (the coordinator is the one place that
  issues hublvol attaches) and again from the deferred redundant-path
  worker, off the critical path, once the second NIC has landed.
- reattach_sibling_failover is the one hublvol attach that bypasses the
  coordinator, so it asserts the policy directly instead of waiting for a
  later reconcile to converge it.
- rpc_client: optional selector / rr_min_io / request_timeout, omitted
  from params when unset so SPDK's defaults apply.

The helper is non-fatal by design, against the CONTRIBUTING preference for
raising: callers gate their rejoin on the attach, not on the policy, and a
missed assertion only degrades the hub to the previous ACTIVE_PASSIVE
behaviour, which is not worth failing a restart over. Every reconcile
re-asserts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…arent

Cross-cluster DR fail-over produced volumes full of zeros: no filesystem,
blkid empty, mount -t xfs failing on a bad superblock, md5 mismatch on every
volume — while the fail-over itself reported success end to end (all five
relationships failed_over, valid connection strings, lvols online). Reproduced
on two labs (2026-08-10, 2026-08-11).

snapshot_monitor.process_snap_delete treated ONLY an in-deletion clone as a
blocker. A healthy clone did not stop the delete at all, so the monitor
hard-deleted the snapshot a fail-over volume was cloned from — the log shows the
five parents removed ~12 min AFTER the clones were created and online, with the
volumes reading zeros from then on. snapshot_controller._delete_locked already
treats a live clone as blocking (it soft-deletes and keeps the blob); the
monitor finalises that same delete, so leaving it weaker simply undid the
protection. A live clone now blocks, re-reading the record first so a clone that
has since gone does not defer the snapshot forever.

The evidence also ruled out two earlier theories, both recorded in the tests so
they stay ruled out: replication was moving real data (transfer offsets to 23 MB,
snapshots with 5 GB used), and the parents were deleted after the clone existed
rather than in the select->clone window.

Also hardens the fail-over clone itself: it selected the last replicated
snapshot and cloned from it as two unsynchronised steps, while
snapshot_controller.delete() documents that a concurrent clone-create "holds the
same lock for its whole sequence" — the normal clone path honours that, this one
did not. It now clones under the snapshot's object_mutation_lock, re-validates
under the lock, and falls back to an older snapshot rather than cloning a parent
that disappeared. That window is narrower than the bug above, but it is real.

Test harness (scripts/test_async_replication.py):
- new case3: online delta fail-back to the recovered primary, fio must not stop
- new case4: full fail-back to a fresh, empty cluster (third cluster in the
  deploy config) — fail-back needs the CURRENT host cluster's add-replication
  repointed, because replication_commit takes the destination POOL from cluster
  config while taking the destination NODE from the volume
- new case5/case6: replication must survive a target node, and a source primary
  (secondary surviving), going offline and coming back — using sn shutdown /
  restart rather than killing SPDK, which the control plane silently auto-heals
- case runner with groups (both | failback | errors | all) that reports every
  case instead of stopping at the first failure

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An lvol's namespace is reported by SPDK under the name its bdev was
registered with — the raw UUID — not the <lvs>/<lvol> alias carried in
lvol.top_bdev. Every comparison that only checked bdev_name therefore read
a present namespace as absent.

That inverted the guard it fed. add_lvol_thread's post-condition ("refusing
to add a listener for an empty subsystem"), added for the 2026-08-09
listener-without-namespace incident, saw no namespace and skipped listener
creation. Multipath soak 2026-08-11, first outage pair (aa60b24a shutdown +
0baf99e4 network_outage, 20.6s offset, overlapping): both recovered nodes
came back with every lvol subsystem holding its namespace and ZERO
listeners. Four of six volumes silently lost a path; the control plane
reported every lvol ha; fio never errored because the primary path still
served IO. health_controller detected it correctly and lvol_monitor's repair
re-refused it on every cycle for the same reason, so it was permanent — the
exact mirror of the incident the guard exists to prevent.

- rpc_client.namespace_matches(): one place that decides namespace identity.
  Matches on the namespace UUID or the bdev name; a contradicting UUID still
  disqualifies a bdev_name match, since the same bdev name carrying a
  different volume is a real conflict rather than a match.
- _rpc_subsystem_has_ns() takes uuid and delegates to it; both call sites in
  add_lvol_thread pass lvol.uuid. This fixes the idempotency check too,
  which previously re-issued add_ns for an already-bound namespace.
- nvmf_subsystem_add_ns()'s duplicate-rejection probe uses the same matcher,
  so a rejected duplicate add is recognised instead of surfacing as failure.
- New _rpc_wait_subsystem_has_ns(): the post-condition polls briefly. add_ns
  can report success just before the namespace is observable, and one read
  is not enough to justify permanently skipping a listener.

Also closes the hublvol multipath-policy window from df11dae:
connect_to_hublvol skips the coordinator entirely when the remote bdev
already exists, so the active_active assertion never ran on that path and a
re-attached hublvol sat at SPDK's ACTIVE_PASSIVE default with one NIC
carrying all hub IO (seen on a non-outaged peer during the same soak, until
a later reconcile healed it). Asserted on the way out, wait=False so the
in-freeze path does not poll.

Soak scripts:
- aws_dual_node_outage_soak_multipath.py rewritten to the intended test:
  fio at iodepth 128 / 4 jobs / 100 GiB / libaio (was iodepth 4, 16 GiB, and
  ioengine "aiolib", which is not an fio engine); a deterministic all-node
  single-NIC phase where any fio blip including a max_latency violation
  fails; then an overlapping dual-node pair at a random 1-60s offset with a
  30s hold before recovery, combining network_outage / shutdown /
  container_kill / host_reboot. Pairs rotate ring distance so subsequent
  nodes and nodes one or two apart get equal coverage. Role topology is read
  live from the CP instead of a metadata "topology" key the deployer never
  writes, which had left the old role-pair exclusion silently empty. The
  concurrent NIC-chaos thread is gone: it overlapped node outages and
  misattributed the resulting errors. Fixed 90s wait between pairs, no
  rebalance wait. Verifies mp_policy=active_active on hublvol bdevs.
- Both soaks: docker exec needs -u root (the container runs as uid 1000, so
  /root/spdk/scripts/rpc.py was unreadable and every verification RPC
  returned nothing), and path counting must sum the ctrlrs entries — SPDK
  reports one entry per NIC with an empty alternate_trids, so the old
  1 + len(alternate_trids) test failed every healthy 2-path controller.
  Neither check had ever actually executed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>